Skip to content

BC5 readiness: TypeScript live wire-capture canary#294

Merged
jeremy merged 9 commits into
mainfrom
bc5-readiness-ts-live-canary
Jul 22, 2026
Merged

BC5 readiness: TypeScript live wire-capture canary#294
jeremy merged 9 commits into
mainfrom
bc5-readiness-ts-live-canary

Conversation

@jeremy

@jeremy jeremy commented May 1, 2026

Copy link
Copy Markdown
Member

Stacked on #293. Merge target: bc5-readiness-spec-foundations. Will be re-targeted to main once #293 lands.

Summary

The TypeScript runner becomes the canonical wire-capturer for the SDK's compatibility canary. It dispatches each live test through the SDK's typed surface, intercepts the raw HTTP response at the global fetch chokepoint, and validates the body against the OpenAPI response schema.

Strictly the TS layer of the multi-language canary plan: cross-language replay and pairwise BC4↔BC5 comparison are deferred to follow-up PRs.

What lands

  • Schema extension (conformance/schema.json): mode: "mock" | "live" (default mock); mockResponses conditionally required; liveAssertions[] with four assertion types; fixtureIds[] for placeholder resolution.
  • Live fixture (conformance/tests/live-my-surface.json): 10 operations covering the read surface BC5 forward-compat additions touch (GetMyNotifications, GetTodoset, GetMyProfile, etc.).
  • Mode gating: runner.test.ts loads only mock tests; live-runner.test.ts loads only live tests, skips entirely without BASECAMP_LIVE=1.
  • Wire capture (wire-capture.ts): wraps globalThis.fetch — single chokepoint for both openapi-fetch (page 1) and the fetchPage closure (page 2+). SDK behavior unchanged.
  • Schema validation (schema-validator.ts): ajv + ajv-formats. additionalProperties permissive (forward-compat doesn't break canary); required strict; $ref resolved against openapi.json. Per-page validation; extras union across pages and surface in the run summary.
  • Fixture-ID resolution (fixtures.ts): env-var → discovery → cache ladder (§5d). BASECAMP_BC4_PROJECT_IDBASECAMP_PROJECT_IDListProjects → first project, same for TODOSET_ID, TODOLIST_ID, TODO_ID.
  • Dispatch coverage gate (live-dispatch.ts): assertDispatchCoverage() fires in beforeAll. Any operation in the fixture without a dispatch case fails the run with a pointer to where to add the mapping.
  • Snapshot persistence: when LIVE_RECORD_DIR is set, per-test snapshots write to <dir>/<backend>/wire/<test>.json for downstream replay (PR 3) and pairwise comparison (PR 4).
  • make conformance-typescript-live: opt-in target; not invoked by make check.
  • CONTRIBUTING.md "Live canary": env vars, snapshot dir layout, fixture/dispatch pairing rule.

Acceptance bar

Per the scope guard:

  • Mode-gated — BASECAMP_LIVE unset → mock only; set → live only.
  • Captures raw response bytes/headers at the global fetch boundary.
  • Validates with Ajv against the generated OpenAPI schema (per page, with extras-observed reporting).
  • Deterministic fixture naming — snapshots are namespaced by BASECAMP_BACKEND.
  • Runs against BC4 or BC5 without doing pairwise comparison yet.
  • Cross-language replay and BC4/BC5 diffing remain out.

Test plan

  • make conformance-typescript (default, no live env) — live-runner.test.ts skipped, runner.test.ts 63 mock tests pass + 5 pre-existing TS_SDK_SKIPS.
  • BASECAMP_LIVE=1 npx vitest run live-runner.test.ts (no token) — fails fast in beforeAll with a clear missing-env-var error.
  • Coverage gate: temporarily added NoSuchOperation to the fixture; runner refused to start with: Live runner is missing dispatch cases for: NoSuchOperation. Add a DispatchFn to LIVE_OPERATIONS in live-dispatch.ts. Reverted before commit.
  • Manual: BASECAMP_LIVE=1 BASECAMP_TOKEN=... BASECAMP_ACCOUNT_ID=... BASECAMP_HOST=... LIVE_RECORD_DIR=tmp/live-canary/bc4 BASECAMP_BACKEND=bc4 make conformance-typescript-live against a real BC4 sandbox. (Requires sandbox creds — reviewer to confirm or defer to follow-up.)

Out of scope (later PRs)

  • PR 3: cross-language wire-replay decoders (Ruby/Python/Go/Kotlin).
  • PR 4: pairwise BC4↔BC5 comparison + make check-bc5-compat orchestrator + scheduled CI.
  • PR 5: API gap detector tooling (scripts/detect-api-gaps.sh).

Summary by cubic

Adds a TypeScript live wire-capture canary that validates raw API responses against the OpenAPI schema, collapses same‑URL retries, pre‑resolves fixtures, and writes snapshots for replay. Live runs are opt‑in; offline runners load only mock tests.

  • New Features

    • Conformance schema adds mode (mock/live), liveAssertions, and fixtureIds with mode‑specific requirements; new conformance/tests/live-my-surface.json (~10 read endpoints).
    • TypeScript live runner wraps globalThis.fetch, validates with ajv + ajv-formats, unions extras in the run summary, and persists snapshots to LIVE_RECORD_DIR/<backend>/wire/*.json; gated by BASECAMP_LIVE=1 with BASECAMP_TOKEN/BASECAMP_ACCOUNT_ID; BASECAMP_HOST is origin‑only and the runner appends /{accountId}; make conformance-typescript-live target and CONTRIBUTING docs added.
    • Dispatch coverage gate uses Object.hasOwn; fixture IDs resolve via env → discovery → cache and are pre‑resolved before capture; Go/Ruby/Python/Kotlin filter to mode: "mock" at load time.
  • Bug Fixes

    • Validator: rewrites local $ref to openapi.json#/…, resolves multi‑level $ref chains, finds non‑200 2xx schemas (e.g., 201), treats bodyless 2xx (204‑only ops) as success and flags bodies there as drift, and collects extras recursively through arrays/objects.
    • Live runner: fixes required‑field checks through arrays with repeated segments, uses testCtx.skip for missing fixtures, keeps discovery traffic out of snapshots, and collapses consecutive same‑URL retries so only the final attempt is recorded.
    • Offline tests cover the validator, wire‑capture, and dispatch gate paths; dependencies bump transitive fast-uri to 3.1.4 to clear advisories.

Written for commit 9a6f062. Summary will update on new commits.

Review in cubic

Copilot AI review requested due to automatic review settings May 1, 2026 22:43
@github-actions github-actions Bot added documentation Improvements or additions to documentation dependencies Pull requests that update a dependency file conformance Conformance test suite enhancement New feature or request and removed documentation Improvements or additions to documentation labels May 1, 2026
@jeremy
jeremy force-pushed the bc5-readiness-spec-foundations branch from 5bd407c to 63205bf Compare May 1, 2026 22:49
@jeremy
jeremy force-pushed the bc5-readiness-ts-live-canary branch from b7e9460 to d776bfd Compare May 1, 2026 22:50
@github-actions github-actions Bot added documentation Improvements or additions to documentation and removed documentation Improvements or additions to documentation labels May 1, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b7e946013c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread conformance/runner/typescript/live-runner.test.ts Outdated
Comment thread conformance/runner/typescript/schema-validator.ts Outdated
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label May 1, 2026

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

6 issues found across 12 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="conformance/runner/typescript/live-dispatch.ts">

<violation number="1" location="conformance/runner/typescript/live-dispatch.ts:106">
P2: Dispatch coverage check should not use `in` on a plain object; inherited keys can bypass missing-operation detection.</violation>
</file>

<file name="conformance/runner/typescript/schema-validator.ts">

<violation number="1" location="conformance/runner/typescript/schema-validator.ts:75">
P2: Response schema lookup ignores non-200 success responses (e.g. 201), causing false validation failures for valid operations.</violation>

<violation number="2" location="conformance/runner/typescript/schema-validator.ts:124">
P1: Compiling a response schema fragment leaves `#/components/...` refs without the OpenAPI root context, which can break AJV `$ref` resolution at runtime.</violation>

<violation number="3" location="conformance/runner/typescript/schema-validator.ts:182">
P2: `readObjectProperties` returns `null` for array response schemas, causing `collectExtras` to short-circuit and return `[]`. List endpoints (ListProjects, ListTodolists, ListTodos, etc.) that return top-level arrays will never report additive fields inside array items, silently missing BC5 forward-compat wire additions in the extras-observed summary. Consider unwrapping `items` for array schemas to inspect item-level properties.</violation>
</file>

<file name="conformance/runner/typescript/live-runner.test.ts">

<violation number="1" location="conformance/runner/typescript/live-runner.test.ts:203">
P2: Use Vitest's skip API here; rejecting the async test will mark missing fixtures as failures instead of skips.</violation>

<violation number="2" location="conformance/runner/typescript/live-runner.test.ts:284">
P1: Extras reporting does not work for array responses, so list endpoints won't surface additive wire fields.</violation>
</file>

Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.

Comment thread conformance/runner/typescript/schema-validator.ts Outdated
Comment thread conformance/runner/typescript/live-runner.test.ts
Comment thread conformance/runner/typescript/live-dispatch.ts Outdated
Comment thread conformance/runner/typescript/schema-validator.ts Outdated
Comment thread conformance/runner/typescript/live-runner.test.ts Outdated
Comment thread conformance/runner/typescript/schema-validator.ts Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0e243c17ef

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread conformance/runner/typescript/live-runner.test.ts Outdated
@jeremy
jeremy force-pushed the bc5-readiness-spec-foundations branch from 63205bf to 7e9c434 Compare May 1, 2026 23:03

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 3 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="conformance/runner/typescript/schema-validator.ts">

<violation number="1" location="conformance/runner/typescript/schema-validator.ts:163">
P2: `collectExtras` resolves `$ref` only one level, so alias chains (common in response schemas) cause valid fields to be misreported as extras.</violation>
</file>

Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.

Comment thread conformance/runner/typescript/schema-validator.ts Outdated
@jeremy
jeremy force-pushed the bc5-readiness-ts-live-canary branch from 0e243c1 to bc9c7e2 Compare May 1, 2026 23:09

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: bc9c7e2b92

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread conformance/runner/typescript/live-runner.test.ts Outdated
@github-actions github-actions Bot added the kotlin label May 1, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 01ea18612f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread conformance/runner/typescript/schema-validator.ts
@github-actions github-actions Bot added documentation Improvements or additions to documentation and removed documentation Improvements or additions to documentation labels May 2, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ebaac5db99

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread conformance/runner/python/replay_runner.py Outdated
Comment thread conformance/runner/ruby/replay-runner.rb Outdated

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

5 issues found across 15 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="kotlin/conformance/src/main/kotlin/com/basecamp/sdk/conformance/SchemaWalker.kt">

<violation number="1" location="kotlin/conformance/src/main/kotlin/com/basecamp/sdk/conformance/SchemaWalker.kt:59">
P2: Include OpenAPI wildcard success code `"2XX"` in the fallback response-schema lookup; otherwise valid schemas can be missed and live responses go unvalidated.</violation>
</file>

<file name="conformance/runner/go/replay_runner.go">

<violation number="1" location="conformance/runner/go/replay_runner.go:252">
P2: Validate that each snapshot’s `operation` matches the fixture test’s expected operation before decoding. Without this check, swapped/mislabeled snapshot files can be decoded against the wrong schema and mask fixture mapping errors.</violation>
</file>

<file name="conformance/runner/ruby/replay-runner.rb">

<violation number="1" location="conformance/runner/ruby/replay-runner.rb:34">
P2: `SDK_DECODE` returns `nil` (success) for empty `body_text`, but the real SDK path calls `JSON.parse(body_text)` which raises `JSON::ParserError` on empty strings. This mismatch means the replay runner treats empty-body snapshots as successful decodes that would actually fail through the real SDK, producing false negatives in compatibility checks. Remove the early return or raise on empty body to match real SDK behavior.</violation>

<violation number="2" location="conformance/runner/ruby/replay-runner.rb:75">
P2: Validate that each snapshot file’s `operation` matches the fixture test’s expected operation; otherwise swapped/mislabeled snapshot files can be silently accepted and validated against the wrong schema.</violation>
</file>

<file name="conformance/runner/python/replay_runner.py">

<violation number="1" location="conformance/runner/python/replay_runner.py:163">
P2: Python's `or` treats empty string `""` as falsy, so when `bodyText` is empty this falls through to `json.dumps(page["body"])`. If `body` is `None`, that produces `"null"`, which `json.loads("null")` parses successfully — masking an actually-empty wire body as a successful decode. Compare with the Go equivalent which guards with `bodyText == "" && page.Body != nil`. Use an explicit `None` check instead of `or`:
`body_text = page.get("bodyText") if page.get("bodyText") is not None else json.dumps(page["body"])`
or handle empty bodyText as a decode boundary separately.</violation>
</file>

Tip: Review your code locally with the cubic CLI to iterate faster.

Comment thread kotlin/conformance/src/main/kotlin/com/basecamp/sdk/conformance/SchemaWalker.kt Outdated
Comment thread conformance/runner/go/replay_runner.go Outdated
Comment thread conformance/runner/ruby/replay-runner.rb Outdated
Comment thread conformance/runner/ruby/replay-runner.rb Outdated
Comment thread conformance/runner/python/replay_runner.py Outdated
@jeremy
jeremy force-pushed the bc5-readiness-ts-live-canary branch from ebaac5d to 149ec69 Compare May 4, 2026 23:50
@github-actions github-actions Bot added documentation Improvements or additions to documentation and removed documentation Improvements or additions to documentation labels May 4, 2026
jeremy added a commit that referenced this pull request May 13, 2026
Both come in transitively via openapi-typescript → @redocly/openapi-core
(fast-uri through @redocly/ajv, brace-expansion through minimatch).
npm audit on the affected versions reports:

  fast-uri ≤3.1.1 — high — path traversal + host confusion
  (GHSA-q3j6-qgpj-74h6 / GHSA-v39h-62p7-jpjc)
  brace-expansion 4.0.0–5.0.4 — moderate — zero-step sequence
  hangs the process (GHSA-f886-m6hf-6m8v)

Lift both to the patched majors via overrides (alongside the existing
minimatch pin). The override path is preferred over plain dependency
bumps because the SDK doesn't depend on either package directly —
the responsibility for forwarding the fix sits with this repo until
the upstream packages cut new releases that pull the patches in
transitively.

Lands at the bottom of the BC5 stack so the wire-replay PRs above
(#294, #301) inherit the fix on rebase.
@jeremy

jeremy commented Jul 14, 2026

Copy link
Copy Markdown
Member Author

Rebased onto the rebased #293 (dce56102, itself on current main) — clean replay, 7 commits preserved — plus one commit (1a401145) clearing all three blockers:

  1. Retry-aware capture (codex P1): wire capture collapses consecutive same-URL captures to the final attempt, so a 429→200 recovery records one page with status 200. Fixed at the capture layer so liveCallSucceeds, pages_count, and pairwise comparison all see retry-free pages. New wire-capture.test.ts drives the real retry middleware with a mocked 429→200.
  2. Bodyless-op drift check (cubic P2): a present body on a spec-bodyless operation now hard-fails naming the operation; absent bodies still pass. Both directions tested.
  3. Dependency Review: fast-uri 3.1.0 → 3.1.3 in the conformance runner lockfile (GHSA-q3j6-qgpj-74h6 / GHSA-v39h-62p7-jpjc).

Conformance TS suite green locally: 84 passed / 6 skipped (mock baseline unchanged).

jeremy added a commit that referenced this pull request Jul 22, 2026
Both come in transitively via openapi-typescript → @redocly/openapi-core
(fast-uri through @redocly/ajv, brace-expansion through minimatch).
npm audit on the affected versions reports:

  fast-uri ≤3.1.1 — high — path traversal + host confusion
  (GHSA-q3j6-qgpj-74h6 / GHSA-v39h-62p7-jpjc)
  brace-expansion 4.0.0–5.0.4 — moderate — zero-step sequence
  hangs the process (GHSA-f886-m6hf-6m8v)

Lift both to the patched majors via overrides (alongside the existing
minimatch pin). The override path is preferred over plain dependency
bumps because the SDK doesn't depend on either package directly —
the responsibility for forwarding the fix sits with this repo until
the upstream packages cut new releases that pull the patches in
transitively.

Lands at the bottom of the BC5 stack so the wire-replay PRs above
(#294, #301) inherit the fix on rebase.
@jeremy
jeremy force-pushed the bc5-readiness-spec-foundations branch from dce5610 to 839b690 Compare July 22, 2026 02:34
@jeremy
jeremy force-pushed the bc5-readiness-ts-live-canary branch from 1a40114 to 7c615c3 Compare July 22, 2026 02:40
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Jul 22, 2026
@jeremy

jeremy commented Jul 22, 2026

Copy link
Copy Markdown
Member Author

Rebased onto the new #293 head (839b690, itself freshly rebased onto main@558cfdc3). Rebase-only — no content changes. Sole conflict was the Makefile .PHONY line (kept both conformance-typescript-live and main's oauth-fixtures-check). Verified: TS conformance suite 90 passed / 5 skipped (live-mode tests skip offline), TS SDK make ts-check 716 passed, npm ls fast-uri resolves ≥3.1.2 (3.1.2/3.1.3).

@github-actions github-actions Bot removed the documentation Improvements or additions to documentation label Jul 22, 2026
@jeremy
jeremy force-pushed the bc5-readiness-ts-live-canary branch from 7c615c3 to cb47f9e Compare July 22, 2026 02:47
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Jul 22, 2026
@jeremy

jeremy commented Jul 22, 2026

Copy link
Copy Markdown
Member Author

Follow-up: #293 advanced again to 12c0f0a (fast-uri/brace-expansion/js-yaml override bumps for fresh advisories), so this branch is now rebased onto that head — zero conflicts. One new commit rides along: conformance(ts): bump runner fast-uri to 3.1.4 for GHSA-v2hh-gcrm-f6hxnpm audit --audit-level=high in conformance/runner/typescript flagged fast-uri <=3.1.3; npm audit fix resolves 3.1.4 within ajv's range and the audit is now clean. Re-verified: conformance suite 90 passed / 5 skipped.

@github-actions github-actions Bot removed the documentation Improvements or additions to documentation label Jul 22, 2026
jeremy added a commit that referenced this pull request Jul 22, 2026
* spec(bc5): forward-compat additions + parallel BC4/BC5 provenance

Add optional fields to existing Smithy structures for BC5 read paths,
keeping BC4 readers fully compatible. All additions are opt-in: absent
fields are tolerated, present fields decode through the existing types.

* Notification: bubble_up_url, bubble_up_at (eligibility-gated)
* GetMyNotificationsOutput: bubble_ups, scheduled_bubble_ups; document
  current memories[] behavior pending BC team's back-compat decision
* Person: tagline (alias of bio in BC5)
* Todo: steps (reuses existing CardStep wire shape)
* Todoset: todos_count, completed_loose_todos_count, todos_url, app_todos_url

Restructure spec/api-provenance.json for parallel branch tracking — .bc3
keeps the active-branch baseline (now five), .compatibility.bc3-master
records the BC4 baseline for compatibility tracking. Existing tooling
(sync-spec-version, sync-api-version, Go provenance.go) reads the same
.bc3 keys and works unchanged.

Seed spec/bucket-scoped-allowlist.txt with the only existing bucket-only
list endpoint (webhooks) for the new parity lint.

* build(make): add generate aggregator, parity lint, branch-aware sync-status

* make generate — single command to regenerate every machine-derived
  artifact in the repo (Smithy, behavior model, URL routes, provenance,
  per-language SDKs) in dependency order.

* make check-bucket-flat-parity — new lint that scans openapi.json for
  GET /{accountId}/buckets/{bucketId}/<resource>(/...).json list
  operations and verifies each has a flat counterpart at
  /{accountId}/<resource>.json (or is justified in
  spec/bucket-scoped-allowlist.txt). Cross-project SDK consumers
  shouldn't have to walk every project to query account-wide data.

* make sync-status — reads .bc3.branch and reports drift for the active
  branch (now five) and the compatibility branch (master) as two
  clearly-labeled blocks via scripts/report-bc3-drift.sh, instead of
  silently mixing them under HEAD.

* make validate-api-gaps target wired into make check (script lands in
  the api-gaps registry commit).

* docs(api-gaps): API gap registry + SDK↔BC3 coordination + validator

The SDK is the client-side demand signal for BC3 API coverage. spec/api-gaps/
is a durable, schema-validated registry: each entry tracks one BC5
user-visible feature that ships without (or with incomplete) JSON API
coverage, paired with the BC3 parity plan that owns server-side delivery.
Status changes flow through git history, making the absorption journey
publicly auditable.

Initial registry (11 entries aligned with BC3 plan Phase 3 deliverables):

* calendar — show/update only
* scratchpad — URL path deferred by BC3 plan
* step-top-level — doc-only addition (existing partial)
* everything-aggregates — Phase 3c flat top-level recording listings (~22)
* activity-timeline — /activity (global) + /projects/:id/timeline
* recordable-subtypes-doc — Journal, CloudFile, GoogleDocument; Door excluded
  (read paths + create operations per BC3 Phase 3a)
* stack-doc-and-smithy — full CRUD + list collectables
* search-filter-additions — additive filter params
* rich-text-project-attachable — Project as Attachable
* recording-bubbleupable-field — additive boolean
* todoset-completed-list-visibility — pending BC3 classification

Tooling:
* spec/api-gaps/schema.json + allowlist-schema.json validate frontmatter
* allowlist.yml records routes that don't warrant a registry entry
  (transient nav state, duplicates already covered elsewhere)
* scripts/validate-api-gaps.{sh,rb} — Ruby stdlib only, no gem deps

COORDINATION.md (root) describes the SDK ↔ BC3 plan relationship and the
absorption lifecycle for anyone reading the SDK repo cold.

CONTRIBUTING.md picks up the api-gaps section, the parity lint section,
and points at make generate as the single command to regenerate
everything.

* ts: regenerate from BC5 spec additions

* ruby: regenerate from BC5 spec additions

* python: regenerate from BC5 spec additions

* kotlin: regenerate from BC5 spec additions

* swift: regenerate from BC5 spec additions

* go: regenerate from BC5 spec additions

* typescript: pin fast-uri and brace-expansion via overrides

Both come in transitively via openapi-typescript → @redocly/openapi-core
(fast-uri through @redocly/ajv, brace-expansion through minimatch).
npm audit on the affected versions reports:

  fast-uri ≤3.1.1 — high — path traversal + host confusion
  (GHSA-q3j6-qgpj-74h6 / GHSA-v39h-62p7-jpjc)
  brace-expansion 4.0.0–5.0.4 — moderate — zero-step sequence
  hangs the process (GHSA-f886-m6hf-6m8v)

Lift both to the patched majors via overrides (alongside the existing
minimatch pin). The override path is preferred over plain dependency
bumps because the SDK doesn't depend on either package directly —
the responsibility for forwarding the fix sits with this repo until
the upstream packages cut new releases that pull the patches in
transitively.

Lands at the bottom of the BC5 stack so the wire-replay PRs above
(#294, #301) inherit the fix on rebase.

* spec: address Copilot review threads on bucket-parity lint + brief

scripts/check-bucket-flat-parity.sh — drop `--slurpfile spec` and read
the spec via positional input instead. The previous form passed the
spec to jq twice (once via --slurpfile, once positionally), so jq
parsed it twice per invocation. Reading once via positional input then
binding `. as \$s` for cross-iteration access preserves the same
schema-resolution behavior with a single parse.

spec/api-gaps/todoset-completed-list-visibility.md — rewrite the
"PR 1 must classify" sentence so the constraint reads on its own,
without requiring the umbrella-plan cross-reference. The brief now
notes that classification must come from the BC3 side and the brief
stays in the registry either way; readers no longer need to know
which PR "PR 1" referred to.

* make: iterate compatibility provenance blocks generically in sync-status

The sync-status target hardcoded the .compatibility.bc3-master key, which
broke when the post-launch repin renamed the BC4 baseline block to
bc3-four (BC4 now lives on the four branch; master is BC5). Loop over
.compatibility | keys[] instead so any set of compatibility baselines
gets its own labeled drift report without further Makefile edits.

* regen: reconcile generated artifacts after rebase onto main

make generate over the hand-merged spec picks up both main's project
schedule dates (#302) and this branch's BC5 forward-compat additions,
and stamps the api-provenance repin (bc3 = master @ 415b54cc, BC4
compatibility baseline = four @ 9d73959a) through the Go provenance
mirror. go.work.sum picks up entries from the go generate dep download.

* typescript: bump fast-uri, brace-expansion, js-yaml overrides for fresh advisories

Three high-severity advisories published since main's last green security
run (2026-07-20) trip npm Audit on the TypeScript SDK:

- fast-uri GHSA-4c8g-83qw-93j6 (host confusion via failed IDN
  canonicalization): override >=3.1.2 -> >=3.1.3, resolves 4.1.1
- brace-expansion GHSA-3jxr-9vmj-r5cp (DoS via exponential expansion):
  override >=5.0.6 -> >=5.0.7
- js-yaml GHSA-52cp-r559-cp3m (quadratic CPU via merge-key chains):
  new override ^4.3.0, pinned to the 4.x patched line since js-yaml 5
  is a major bump for openapi-typescript's transitive use

npm audit clean; ts-generate + ts-generate-services + ts-check verified
green under the new resolutions.

* make, scripts: address review threads on sync-status + parity lint

- sync-status: pass BC3_REPO through to report-bc3-drift.sh in both the
  primary and compatibility invocations — make variables aren't exported
  to recipe environments, so 'make sync-status BC3_REPO=org/fork' was
  silently querying the default repo
- parity lint: use [[:space:]] instead of the non-POSIX \s escape in the
  allowlist filter; accept {projectId} alongside {bucketId} in the
  bucket-path pattern (and strip logic) for forward-compat with the
  documented Smithy path convention; use # as the sed delimiter to keep
  the ERE alternation unambiguous across GNU/BSD sed

Candidate count and lint verdict unchanged (verified against openapi.json
before/after).

* make, scripts: second review round on sync-status + parity lint

- parity lint: accept literal filter segments (e.g. /todos/completed.json)
  in addition to path-parameter variants — the header always promised
  (/<filter>)? coverage but the regex only matched /{param}. Candidate
  set unchanged today (verified: single-resource GETs carry no .json
  suffix in this spec, so the anchor keeps the match tight).
- sync-status: guard on jq alongside the existing gh guards.
Base automatically changed from bc5-readiness-spec-foundations to main July 22, 2026 04:38
jeremy and others added 9 commits July 21, 2026 21:41
* mode: "mock" | "live" (default mock)
* mockResponses required when mode=mock; forbidden when mode=live
* liveAssertions[] with liveCallSucceeds, liveResponseFieldsRequired,
  liveResponseFieldsExpected, liveSchemaValidate
* fixtureIds[] for placeholder resolution (PROJECT_ID, TODOSET_ID, etc.)

Initial live fixture (live-my-surface.json) covers the read surface
exercised by the BC5 forward-compat additions: ListProjects, GetProject,
GetMyAssignments + completed + due variants, GetMyNotifications,
GetMyProfile, GetTodoset, ListTodolists, ListTodos.

Live tests are gated by BASECAMP_LIVE=1 and never reach MSW in mock-mode
runs (per the runner's mode filter, separate commit).
The TypeScript runner becomes the canonical wire-capturer for the SDK's
compatibility canary: it dispatches each live test through the SDK's typed
surface, intercepts the raw HTTP response (bytes + headers) at the global
fetch chokepoint, and validates the body against the OpenAPI response
schema. Forward-compat fields surface as "extras observed" — never as
failures — so new BC5 fields don't break the canary while staying visible.

Mode gating
* runner.test.ts loads only mode=mock tests; live entries never reach MSW.
* live-runner.test.ts loads only mode=live tests; entire suite skips
  unless BASECAMP_LIVE=1. Missing token/account-id error in beforeAll
  with a clear message before any test runs.

Wire capture
* wire-capture.ts wraps globalThis.fetch — single chokepoint for both
  openapi-fetch (page 1) and the fetchPage closure (page 2+). Cloned
  responses preserve raw bytes for snapshot persistence; SDK behavior
  unchanged.
* Snapshots persist to <LIVE_RECORD_DIR>/<backend>/wire/<test>.json when
  the env var is set — feeds cross-language replay (PR 3) and pairwise
  BC4↔BC5 comparison (PR 4).

Schema validation
* schema-validator.ts uses ajv + ajv-formats. additionalProperties
  permissive (forward-compat must not break the canary); required strict;
  $ref resolved against openapi.json.
* Per-page validation; extras union across pages and report in the
  run-end summary.

Fixture-ID resolution
* fixtures.ts walks the env-var → discovery → cache ladder per §5d:
  BASECAMP_BC4_PROJECT_ID / BASECAMP_BC5_PROJECT_ID → BASECAMP_PROJECT_ID
  → ListProjects → first project. Same pattern for TODOSET_ID,
  TODOLIST_ID, TODO_ID. Missing-fixture path skips with skipReason.

Dispatch coverage gate
* live-dispatch.ts maps operation → SDK call. assertDispatchCoverage()
  fires in beforeAll: any operation referenced in the fixture without a
  dispatch case fails the run with a clear error pointing at where to
  add the mapping.

Make + docs
* `make conformance-typescript-live` opt-in target (not invoked by
  `make check`). CONTRIBUTING.md documents env vars and the
  fixture/dispatch pairing rule.

Out of scope per PR 2: cross-language wire replay (PR 3), pairwise
BC4↔BC5 comparison (PR 4), API gap detector (PR 5).
Three fixes from PR review against the live-only path that offline
checks didn't exercise:

1. Schema validator $ref resolution. The validator registered the full
   OpenAPI doc with key "openapi.json" but compiled the response-schema
   fragment directly. Refs like "#/components/schemas/ListProjectsResponseContent"
   resolved against the fragment root (not the registered doc), throwing
   "can't resolve reference ... from id #" on every live call. Fix:
   prepareForCompile rewrites local refs ("#/...") to fully-qualified
   refs ("openapi.json#/...") so Ajv resolves against the registered
   document. additionalProperties:false is still stripped in the same
   pass.

2. Vitest-native skip for fixture-missing. The previous code did
   Promise.reject(Object.assign(new Error("SKIP..."), { skip: true })),
   which Vitest reports as a failure regardless of any custom flag. A
   sandbox missing a todoset/todolist would have failed the canary
   despite the documented "skip with skipReason" behavior. Fix: use the
   test-context's runtime skip — testCtx.skip(reason) — so missing
   fixtures land as proper skipped tests.

3. Recursive extras collection. collectExtras() only inspected the root
   when it was an object, so list responses (ListProjects, ListTodos,
   etc.) yielded no extras even when items had unknown fields. Fix:
   walk body + schema together, descend through arrays (path segment
   "[]") and into known properties; emit dotted paths like
   "[].future_field" or "unreads[].future_envelope_field". Bounded
   depth as a cycle guard.

Offline coverage: schema-validator.test.ts exercises validateResponse
with crafted payloads — conformant ListProjects/Project, missing
required field, forward-compat extras at root and item level, plus
GetMyNotifications object-with-array properties. 10 tests, all pass.
This is the regression bar for the live-only path: future schema
wiring bugs surface on every PR via mock conformance, no live creds
needed.

Verified: BASECAMP_LIVE=1 BASECAMP_TOKEN=stub BASECAMP_HOST=<unreachable>
runs report 4 fixture-needing tests as skipped (ctx.skip path) and
6 host-needing tests as expected-failed (network). No false failures.
Four issues from the cubic/codex review pass on the live-only path:

1. Schema lookup ignored non-200 success responses. findResponseSchema()
   only checked "200" then "default", missing operations that return 201
   (Create*) or 204 (Delete*). Now scans 200/201/202/203/204/default
   first, then any 2xx code, before declaring "no schema found".

2. resolveRef only resolved one level. Alias chains
   (Foo → $ref Bar → $ref Baz) caused valid item-level fields to be
   misreported as extras. Now loops until non-ref or cycle detected.

3. assertDispatchCoverage used `op in LIVE_OPERATIONS`. Inherited
   Object.prototype keys (`toString`, `hasOwnProperty`, `constructor`)
   would have slipped past the gate. Switched to Object.hasOwn().

4. Wire capture window included fixture-discovery traffic. dispatch(ctx)
   would call resolveFixtureId() inside the capture window, mixing the
   ListProjects discovery page into the snapshot for GetProject. Split
   the dispatch contract into DispatchSpec { fixtures, call }: the
   runner pre-resolves all required fixtures before installing capture,
   then runs the call() with already-resolved IDs. Discovery happens
   strictly before capture starts, so snapshots contain only the
   operation under test.

Offline coverage:
* schema-validator.test.ts gains a "non-200 success" case asserting
  CreateProject (201) finds a schema (vs the pre-fix "no schema found").
* live-dispatch.test.ts (new) covers the coverage gate including the
  Object.hasOwn semantics — fixture entries for `toString` /
  `hasOwnProperty` / `constructor` are correctly rejected.

15 offline tests pass; full TS conformance: 78 passed, 6 skipped.
The live-canary fixture (conformance/tests/live-my-surface.json) sits in
the shared conformance/tests directory but only the TS runners know
about the `mode` field. Without this fix:

* Kotlin parses every JSON in the directory; ${PROJECT_ID} placeholders
  fail Int conversion and live-only operations are unknown — 9 hard
  failures in CI.
* Go has the same loader pattern but its assertion checker silently
  ignores entries lacking `assertions` (live entries use `liveAssertions`),
  so the 10 live entries slip through as false passes (78/0 vs the
  expected 68/0).
* Ruby and Python share the same shape; same risk.

Make `mode` a cross-runner contract: the four offline runners (Go,
Ruby, Python, Kotlin) now filter `mode == "live"` entries at load time.
TS runners already filtered. Schema stays shared in conformance/tests/;
no fixture relocation needed.

Verified with `make conformance` (each runner separately):
* Go: Passed: 68, Failed: 0
* Ruby: 59 passed, 9 SDK-specific skips
* Python: 64 passed, 4 SDK-specific skips
* Kotlin: Passed: 58, Skipped: 10
* TS: 3 mock test files passed, live-runner.test.ts skipped without BASECAMP_LIVE

Each runner now reports 68 total (10 fewer than the pre-fix 78 — exactly
matching the live-only entries that now correctly filter out).
…chema

Ruby was the only offline runner expressed as "reject live" instead of
"accept mock". With the schema enum bounded to {mock, live} the two are
equivalent today, but accept-mock defends against any future mode (e.g.
wire-replay) silently leaking into the offline runner.

Also document in CONTRIBUTING.md that mode is a shared-schema contract
across runners — fixtures live in conformance/tests/ for everyone, so
offline runners must filter to mock before dispatch.
live-runner.test.ts `fieldExists` — when descending into an array, the
remaining-path computation used `parts.indexOf(part)`, which returns
the first occurrence of that segment name. For required-field paths
with repeated names (`a.b.b.x`), recursion restarts from the wrong
position and reports false missing-field failures. Iterate by index
instead.

schema-validator.ts `validateResponse` — when `findResponseSchema`
returns null, the validator returned `ok: false` unconditionally.
That broke schema validation for legitimately bodyless 2xx operations
(e.g. `DeleteBoost` with only a 204 response), blocking them from
inclusion in the live canary. Distinguish bodyless-by-design (`ok: true`,
nothing to validate) from missing-operation (`ok: false`) via a new
`operationHasBodylessSuccessOnly` helper that checks whether the
operation declares any 2xx response without an `application/json`
schema. New regression test pins the `DeleteBoost` case.
…-uri bump

Three review blockers on the live canary:

* Wire capture now collapses consecutive captures of the same URL into
  the final attempt. The SDK's retry middleware re-issues the identical
  request through global fetch on 429/503, so a recovered call recorded
  the failed attempt as pages[0] — failing liveCallSucceeds despite the
  call succeeding, and polluting pages_count/pairwise comparison with
  retry noise. Distinct pagination pages keep distinct URLs, so only
  retry attempts collapse. Covered at the capture layer directly and
  through the real retry middleware (mocked 429 → 200; one page,
  status 200, calls seen = 2).

* validateResponse no longer waves through a present body on operations
  whose spec declares only bodyless 2xx responses. A payload where the
  spec promises none is contract drift; succeeding masked 204-style
  operations growing undocumented bodies. Absent bodies (null/undefined/
  empty string) still pass. Both directions unit-tested.

* conformance runner lockfile: fast-uri 3.1.0 → 3.1.3 (transitive via
  ajv) clearing GHSA-q3j6-qgpj-74h6 / GHSA-v39h-62p7-jpjc, which failed
  the Dependency Review check.
npm audit --audit-level=high flagged fast-uri <=3.1.3 (host confusion via
literal backslash authority delimiter). npm audit fix resolves 3.1.4 within
ajv's ^3.0.1 range; audit is clean. Companion to the SDK-side override bumps
on the spec-foundations branch.
@jeremy
jeremy force-pushed the bc5-readiness-ts-live-canary branch from cb47f9e to 9a6f062 Compare July 22, 2026 04:42
@github-actions github-actions Bot added documentation Improvements or additions to documentation and removed documentation Improvements or additions to documentation labels Jul 22, 2026
@jeremy
jeremy merged commit b196938 into main Jul 22, 2026
45 checks passed
@jeremy
jeremy deleted the bc5-readiness-ts-live-canary branch July 22, 2026 04:49

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9a6f062879

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

capture.restore();
}

const snapshot = capture.drain();

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Verify live captures against fixture routes

After draining the capture, the live runner starts evaluating assertions without checking the captured request URL/method against tc.path/tc.method; this already lets the new GetProject and GetTodoset fixtures declare ...${ID}.json paths even though the generated clients dispatch /projects/{projectId} and /todosets/{todosetId} (no label suffix). A live run can therefore pass while snapshotting a different route than the fixture claims, so please resolve the fixture placeholders and assert the captured route (or remove/correct the unused path metadata).

AGENTS.md reference: AGENTS.md:L178-L183

Useful? React with 👍 / 👎.

jeremy added a commit to robzolkos/basecamp-sdk that referenced this pull request Jul 22, 2026
…reate-bucket-scope-20260601145434

* origin/main: (28 commits)
  Upstream sync: repin provenance to bc3 ba105ba7 (2026-07-22), register post-train contracts, codify pin semantics (basecamp#390)
  ci: pin conformance uv sync to the matrix Python; parse the fan-in gate (basecamp#388)
  BC5 readiness: pairwise BC4↔BC5 + check-bc5-compat orchestrator + scheduled CI (basecamp#308)
  CI: fold conformance suites into their language test jobs (basecamp#387)
  Fix TypeScript SDK Content-Type on GET requests (basecamp#384)
  BC5 launch reconciliation: registry refresh for the shipped BC5 API train (basecamp#325)
  BC5 readiness: cross-language wire-replay decoders (basecamp#301)
  BC5 readiness: Go wrapper forward-compat audit + field-level drift check (basecamp#309)
  BC5 readiness: TypeScript live wire-capture canary (basecamp#294)
  BC5 readiness: forward-compat spec, briefs, parity lint (basecamp#293)
  deps(ts): bump brace-expansion from 5.0.2 to 5.0.7 in /typescript (basecamp#386)
  typescript: bump fast-uri, brace-expansion, js-yaml overrides for fresh advisories (basecamp#385)
  deps(actions): bump the actions group with 6 updates (basecamp#382)
  deps(ruby): bump the bundler-dependencies group in /ruby with 2 updates (basecamp#379)
  deps(kotlin): bump the gradle-dependencies group (basecamp#383)
  deps(go): bump github.com/oapi-codegen/oapi-codegen/v2 in /go (basecamp#381)
  deps(ts): bump the npm-dependencies group in /typescript with 2 updates (basecamp#380)
  deps(smithy): bump the smithy-gradle-dependencies group (basecamp#378)
  deps(go): bump github.com/oapi-codegen/runtime (basecamp#377)
  Todos Update/Edit/Replace: merge-safe writes across all 6 SDKs (basecamp#375)
  ...

# Conflicts:
#	go/pkg/basecamp/api-provenance.json
#	go/pkg/basecamp/version.go
#	go/pkg/generated/client.gen.go
#	kotlin/sdk/src/commonMain/kotlin/com/basecamp/sdk/BasecampConfig.kt
#	openapi.json
#	python/src/basecamp/_version.py
#	ruby/lib/basecamp/generated/metadata.json
#	ruby/lib/basecamp/generated/types.rb
#	ruby/lib/basecamp/version.rb
#	spec/api-provenance.json
#	spec/basecamp.smithy
#	swift/Sources/Basecamp/BasecampConfig.swift
#	typescript/src/client.ts
#	typescript/src/generated/metadata.ts
#	typescript/src/generated/openapi-stripped.json
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

conformance Conformance test suite dependencies Pull requests that update a dependency file enhancement New feature or request kotlin

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant